A Dynamically Reconfigurable Model for a Distributed Web Crawling System

A web crawling system using a distributed architecture needs to coordinate the whole system when the nodes in the system change. This paper presents an efficiently dynamic reconfigurability model that can be used in such a system. Through analyzing the model, we got methods to achieve the optimized performance in the distributed web crawling system, i.e., retain load balance and produce low network traffic in the system. Currently this dynamic reconfigurability model is being introduced in perfecting WebGather, a well-known Chinese and English web search engine. In addition, we believe that the model can also be useful in other web crawling system adopting a distributed architecture.